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BACKGROUND OF THE INVENTION 
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Field of the Invention 

This invention relates to audio outputs in handheld 
devices. In particular, the invention relates to the handling 
of simultaneous audio outputs. 

Prior Art 



Over the years, many electronic devices have been 
sufficiently reduced in size to become portable, and in turn, 
U small enough to be held and operated in the hand of the user. 

P The advent of integrated circuits greatly increased the 

kg I 

L 15 number of devices that could achieve portability and handheld 
fl status, as well as increasing the functionality of such 

devices. 



Initially, handheld devices such as pagers, telephones 
20 and radios were single purpose devices that had a single 
audio source and a single audio output. For such devices, 
there was no contention or confusion between multiple audio 



sources, 
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As handheld devices became more sophisticated, they 
began to have multiple audio sources. Audio sources can be 
divided into two classes: signal events and continuous 
sources. In previous devices with multiple audio sources, the 
sources were usually either signal events (e.g. ringers or 
alarms), or continuous sources (e.g. speech or music), but 
not both. The audio output of such devices with multiple 
sources was obtained by prioritizing the inputs and selecting 
among them, or not prioritizing the inputs and simply 
combining them. 

For purposes of this disclosure, prioritizing of audio 
sources is defined as establishing a ranking of relative 
importance between audio sources. Balancing of an audio 
output is defined as establishing a predetermined relative 
level between at least two audio sources that are present in 
an audio output. 

Although telephones utilize both transient and 
continuous sources, they are typically mutually exclusive in 
use. For a service such as call waiting, a signal event may 
be introduced to the audio output of the receiving party's 
device by superimposing it on the conversation that prompted 
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the call waiting signal, or by blanking the conversation. 
This is an example of simple mixing of two sources. Simple 
mixing is tolerable in telephones since the signal event 
duration is short and any loss of conversation can usually be 
recovered by an inquiry to the other party in the 
conversation that was interrupted. However, when a signal 
event is simply mixed with a source such as a radio 
broadcast, information lost through the interruption is 
usually not recoverable. 

As the functional capabilities of handheld devices has 
increased, the number of audio sources that may be 
incorporated in a single device has expanded. The development 
of low cost memory and efficient compression algorithms has 
lead to a new generation of devices that are capable of 
providing both entertainment and communications functions. An 
example of such a device is a telephone that is capable of 
playing MP 3 files* 

The number of available audio sources in handheld 
devices has increased the probability that two sources may 
interfere. For example, a telephone having a stored music 
playback capability may ring while the music is playing. 
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Typically, in the case of simultaneous signal event and 
continuous source combinations, the solution has been either 
to mix the sources or temporarily suppress the continuous 
source. Mixing of the two sources can render the information 
contained in one or both sources unintelligible. For example, 
if a music program is too loud, a signal event such as a 
phone ring or low battery alarm may not be heard. The 
suppression of one source results in the information from 
that source being lost during the period of suppression. 



The combination of two continuous sources in a handheld 
device is usually dealt with by selecting one or the other, 
as in a radio/tape player. This is an example of prioritizing 

the inputs and producing a single output. However, there are 

y • 

■ ' 15 situations when it would be desirable to combine two 

continuous sources and prioritize them with respect to the 
combined output. 

For example, a device combining a wireless baby monitor 
20 and an FM band radio would have two continuous audio sources. 
The baby monitor signal could be received at the 2.4 GHz ISM 
band, and the radio signal at the commercial FM band. As a 
normal condition, the two sources would be combined so that 
the net effect is that of being in a single room with a baby 
25 and a radio. However, it would be desirable to intelligently 
control the relative sound levels to guarantee that the baby 
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could be heard when crying, regardless of how loud the radio 
program was playing. 
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SUMMARY OF THE INVENTION 



Thus, a need exists for a method of prioritizing and 
balancing simultaneous audio outputs in a handheld device 
while minimizing overall information loss. There is also a 
need for a device that can prioritize multiple continuous 
audio sources in addition to prioritizing a continuous audio 
source and a signal event audio source. 

Accordingly, it is an object of the present invention to 
provide a method of prioritizing two or more audio sources 
and balancing the combined output to minimize information 
loss. It is a further object of the present invention to 
provide a capability to monitor and prioritize two continuous 
audio sources and balance them in a combined output. It is 
also an object to provide flexibility in programming the 
prioritization and balancing. These and other objects and 
advantages of the present invention and others not 
specifically recited above will be described in more detail 
herein. 

A system and method for prioritizing multiple audio 
sources and balancing them in a single combined output in a 
handheld device are disclosed. Audio sources are sensed and 
combined with the level of each source subject to adjustment 
by an attenuator/amplifier. A priority logic unit controls 
the attenuator/amplifier associated with each source. The 
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level of each source being combined is adjusted on the basis 
of the character of the sources being combined and a set of 
programmed rules for prioritization. 

In one embodiment of the present invention, two audio 
sources A and B are sensed by a priority logic unit. Source A 
is a continuous audio source and source B is either a 
continuous audio source or a signal event audio source- The 
two sources are combined into a single output with each 
source having a predetermined level of attenuation or gain 
and thus a predetermined signal level ratio. Upon sensing an 
increase in amplitude of source B above a preset threshold 
level, the attenuation or gain of one or both sources is 
adjusted such that a new signal ratio is established between 
the two sources* 

For example, if source A is a high priority source (e.g. 
a telephone ring or other alert tone) and source B is a lower 
priority source (e.g. a music program) then the sound 
management system may lower the volume on source B , combine 
with source A and output the result. Alternatively, source A 
may be raised in volume, combined with source B, and then 
output. In one embodiment, the sound management system is 
integrated with a palm sized handheld computer system. 



PALM3639/ACM/MDM 



7 



BRIEF DESCRIPTION OF THE DRAWINGS 



The accompanying drawings, which are incorporated in and form 
a part of this specification, illustrate embodiments of the 
invention and, together with the description, serve to 
explain the principles of the invention: 

Figure 1A illustrates a representative handheld device. 

Figure IB illustrates exemplary circuitry of a portable 
computer system. 

Figure 2 shows a functional block diagram for a system 
in accordance with an embodiment of the present claimed 
invention . 

Figure 3 shows a method of prioritizing a signal event 
audio source over a continuous audio source in accordance 
with an embodiment of the present claimed invention. 

Figure 4 shows a method of prioritizing a signal event 
audio source and a continuous audio source in accordance with 
an embodiment of the present claimed invention. 

Figure 5 shows a method of prioritizing two continuous 
audio sources in accordance with an embodiment of the present 
claimed invention. 
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DETAILED DESCRIPTION OF THE INVENTION 



In the following detailed description of the present 
invention, a system and method for prioritizing and balancing 
a combined audio output, numerous specific details are set 
forth in order to provide a thorough understanding of the 
present invention. However, it will be obvious to one skilled 
in the art that the present invention may be practiced 
without these specific details. In other instances well known 
methods, procedures, components, and circuits have not been 
described in detail so as not to unnecessarily obscure 
aspects of the present invention 

Figure 1A shows a representative handheld device 100 
that is sound enabled. Audio sources that may be incorporated 
into such a handheld device include radio frequency and 
infrared reception of analog and digital signals. These 
signals may be commercial broadcast, telephone calls or data 
transmitted from a wireless network. BLUETOOTH is a 
particular short range wireless technology that can link a 
telephone, a radio, or other device to a digital data source. 

In addition to received signals, the handheld device 100 
of Figure 1A may also be capable of generating signal events 
such as an alarm associated with a clock, timer, thermometer, 
or phone battery monitor. The device may also have internal 
audio sources such as speech or music stored in digital or 
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analog form. Fixed or removable media may include magnetic, 
optical, and integrated circuit storage media. The magnetic 
media may be used for analog or digital storage. 
Alternatively, the speech or music material may be stored on 
5 a digital storage medium such as flash memory or random 
access memory (RAM) . Speech or music may be stored in a 
variety of formats such as MP3 for music or Adaptive 
Differential Pulse Code Modulation (ADPCM) for speech. 



W s 



10 Figure IB illustrates exemplary schematic circuitry of 

portable computer system 100. Computer system 100 includes an 
address/data bus 99 for communicating information. It should 
be understood that the general bus 99 may consist of a number 
■50 of buses with different data path widths and operating 

U1 15 frequencies, coupled by bridge circuits. The computer system 
p 100 may or may not contain all of the functional blocks 

yj shown. Computer system 100 includes a central processor 101 

Ft coupled with the bus 99 for processing information and 

instructions, a volatile memory 102 (e.g., random access 
20 memory RAM) coupled with the bus 99 for storing static 

information and instructions for the central processor 101 
and a non-volatile memory 103 (e.g., read only memory ROM) 
coupled with the bus 99 for storing static information and 
instructions for the processor 101. Computer system 100 also 
25 includes an optional data storage device 104 (e.g., thin 

profile removable memory) coupled with the bus 99 for storing 
information and instructions. Device 104 can be removable, 
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such as a flash memory card. As described above, system 100 
also contains a display device 105 coupled to the bus 99 for 
displaying information to the computer user. 

Also included in computer system 100 of Figure IB is an 
alphanumeric input device 106. Device 106 can communicate 
information (spatial data and pressure data) and command 
selections to the central processor 101. 

System 100 also includes an optional cursor control or 
directing device 107 coupled to the bus for communicating 
user input information and command selections to the central 
processor 101. In one implementation, device 107 is a touch 
screen device (also a digitizer) incorporated with screen 
105. Device 107 is capable of registering a position on the 
screen 105 where the stylus makes contact and the pressure of 
the contact. The digitizer can be implemented using well 
known devices, for instance, using the ADS-784 6 device by 
Burr-Brown that provides separate channels for spatial stroke 
information and pressure information. 

The display device 105 utilized with the computer system 
100 may be a liquid crystal display (LCD) device, cathode ray 
tube (CRT) , field emission display (FED, also called flat 
panel CRT) or other display device suitable for creating 
graphic images and alphanumeric characters recognizable to 
the user. Any of a number of display technologies can be 
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used, e.g., LCD, FED, plasma, etc., for the flat panel 
display 105. In one embodiment, the display 105 is a flat 
panel multi-mode display capable of both, monochrome and 
color display modes. 

5 

Signal communication device 108, also coupled to bus 99, 
can be a serial port or USB port. In addition to device 108, 
wireless communication links can be established between 
device 100 and a host computer system (or another portable 
10 computer system) using a BLUETOOTH wireless device 111, an 
infrared device 110. Device 100 may also include a wireless 
receiver device 109. 

In one implementation, the Mobitex wireless 
15 communication system may be used to provide two way 
communication between system 100 and other networked 
computers and/or the Internet via a proxy server. In other 
embodiments, TCP protocol can be used or SMS (Short Message 
Service) can be used. System 100 of Figure IB may also 
20 contain batteries for providing electrical power. Replaceable 
cells or rechargeable batteries can be used. Well known 
electronics may be coupled to the battery to detect its 
energy level and this information can be sampled by the 
processor 101. 

25 

Several of the devices shown in Figure IB may serve as 
audio sources. Audio program material may be stored in RAM 
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120, ROM 103, and Data Storage Device 104. Additionally, 
serial port 108, the wireless receiver 109, the infrared 
device 110, the BLUETOOTH device 111 are capable of receiving 
audio program material in analog or digital formats. A 
5 dedicated audio source 112 may be an audio synthesizer or an 
input for a microphone or other audio input. Also, the 
processor 101 may also serve as a digital audio synthesizer 
in conjunction with appropriate software. 

10 Figure 2 shows a system embodiment of the invention 

involving two sources, audio source A 200 and audio source B 
201. Although the principles of the invention are applicable 
to systems having more than two sources, the embodiment shown 
provides two sources in order to describe the fundamental 

15 operation of the invention. 

Audio source A 200 and audio source B 201 may be either 
a signal event source or a continuous source. A signal event 
source is an audio signal of short duration such as a 

20 telephone ring or an alarm. A signal event is an audio signal 
that is intended to alert a user to a change in state or 
condition and is triggered by the change in state or 
condition. A continuous source is an audio source with a 
typically longer duration than a signal event source, and is 

25 usually not dependent upon an external event, but is selected 
arbitrarily by the user. Audio source A 200 is coupled to a 
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variable attenuator/amplifier 203 and audio source B 201 is 
coupled to a variable attenuator/amplifier 204. 

The variable attenuator/amplifier is typically used to 
5 provide gain to a low level signal and thus act as an 

amplifier. In some cases, where a high level input is used, 
it may act as an attenuator. The attenuation and 
amplification may be done by analog or digital techniques and 
may combine analog to digital conversion or digital to analog 
10 conversion. The variable attenuator/amplifier may be 
O implemented entirely in software. 

\i 

|I Referring again to Figure 2, a priority logic unit 202 

IP is coupled to audio source A 200 and audio source B 201. At a 

J" 15 minimum, the priority logic unit 202 is capable of sensing 

the amplitude and/or presence of each audio source. 
J Optionally, the priority logic unit 202 may have a memory 

hang 

0 buffer that is capable of storing an interval of the signal 

being sensed. The priority logic unit 202 is coupled to 
20 variable attenuator/amplifier 203 and is also coupled to 
variable attenuator/amplifier 204. The priority logic unit 
202 is furnished with a set of prioritization rules allowing 
it to control the amount of gain or attenuation that is 
applied to audio source A 200 by variable 
25 attenuator /amplifier 203, and the amount of gain or 

attenuation that is applied to audio source B 201 by variable 
attenuator/amplifier 204 . 
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There are many possible prioritization rules that can be 
used to prioritize the two audio sources. As an example/ 
Audio source A 200 may be attenuated to the point of being 
effectively muted whenever Audio source B 201 is active (i.e. 
has a non-zero amplitude) . 

For each possible audio source pair in the handheld 
device, the prioritization rules establish the relative gain 
applied to the sources, the absolute levels may be set in 
relation to a fixed decibel level , or it may be referenced to 
the level or presence of one of the sources. Alternatively, 
prioritization between a signal event and a continuous source 
may be dependent upon the presence of a continuous source, 
regardless of the immediate sound level. For instance a .WAV 
file or .MP 3 file that is being played. The prioritization 
rules establish a hierarchy of priority for the audio sources 
in the handheld device. The rules may be implemented in 
hardware or software, or both. 

The output of variable attenuator/ amplifier 203 and 
variable attenuator /amplifier 204 are coupled to a mixer 205 
that combines the two signals into a single output 206. The 
output 206 may be coupled to a speaker, headphone jack, or 
line out. The output 206 may consist of a single channel of a 
stereophonic pair, in which case, a handheld device may have 
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more than one implementation of the system shown in Figure 2, 
as in two channels forming a stereophonic pair. 

It should be noted that signal path 
A 207 and signal path B 208 may carry an analog signal, 
digital signal, or a piecewise combination of both, with 
analog/digital (A/D) or digital/analog (D/A) conversion being 
applied at any point in the signal paths between audio source 
A 200 and audio source B 201, and the output 206. The 
priority logic unit may be a digital device, but is capable 
of performing A/D and D/A conversion as required in order to 
provide source sensing and control of the variable 
attenuator/amplifiers . 

For example, if source A is a high priority source (e.g. 
a telephone ring or other alert tone) and source B is a lower 
priority source (e.g. a music program) then the sound 
management system may lower the volume on source B , combine 
with source A and output the result. Alternatively, source A 
may be raised in volume, combined with source B, and then 
output. In one embodiment, the sound management system is 
integrated with a palm sized handheld computer system. 

Figure 3 shows a particular embodiment of the invention 
wherein audio source B is a signal event source that has a 
higher priority than a continuous audio source A. At time T 0 
Mixer input A has an arbitrary initial level A lt and audio 
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source B is inactive with Mixer input B having a value of 
zero. Mixer input A and Mixer input B are the signals derived 
from audio source A 200 and audio source B of Figure 1, and 
have been processes by the variable attenuator/amplifiers in 
5 concert with the priority logic unit 202. In this scenario, 
Input A could be derived from a continuous source such as a 
radio broadcast, and Input B could be derived from a signal 
event input such as a telephone ringer. At time T lf audio 
source B becomes active and Mixer input B has a level of Bj. 
10 In response to audio source B becoming active, the system 

« causes Mixer input B to be reduced to an attenuated level A A . 

2 A time T 2 audio source B becomes inactive and Mixer input A is 

/? restored to it previous level A z . 

y ? 

ill 15 The prioritization scenario of Figure 3 is particularly 

O useful when the user of the handheld device has the volume of 

til 

Ul the radio broadcast set relatively high, and there is a 

Q likelihood that the telephone ring may be drowned out. At 

iLs. 

lower listening levels, the telephone ring may be combined 
20 with the radio program with adjusting the level of the radio 
program. In this case the prioritization rule applied calls 
for a particular gain ratio to make sure that the ring is 
audible over the music, and if the ring cannot be 
sufficiently amplified the radio program volume is reduced. 
25 In a variation of this rule, the radio program volume may 

always be reduced if the user is using a headphone output in 
order to prevent possible discomfort from too much 
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amplification of the telephone ring. A variation in the 
prioritization rules such as this may be programmable in that 
the priority logic unit detects the nature of the output 
being used (e.g. headphone or speaker) and the source 
selection (e.g. tape, radio or microphone) and adjusts the 
prioritization rules accordingly. The user may have the 
opportunity to establish prioritization rules by means of an 
urgent or non-urgent setting for the telephone ring. 

Figure 4 shows the effects of a non-urgent setting for a 
telephone ring. The timing shown in Figure 4 is the same as 
that shown in Figure 3; however , since mixer input A (radio 
program) now has a higher priority, the level is unchanged 
when the audio source B (telephone ring) becomes active, and 
the ring level is set at a level below the radio program, 
based upon the gain ratio associated with the prioritization 
rule, and referenced to the radio program level. 

Figure 5 shows a scenario in which both sources are 
continuous sources. In this particular embodiment, mixer 
input A is derived from a radio program and audio source B is 
a wireless baby monitor. Both sources are initially of equal 
priority, and they are combined to produce the output. At T 0 
mixer input A is at an arbitrary initial level A z . Between T 0 
and T x the baby' s room is quiet and the mixer output consists 
essentially of mixer input A at the initial level A T and the 
silence in the baby's room. Between T 1 and T 2 , there is a 
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slight noise in the baby's room that is below the threshold 
value B T . Since the noise is below the threshold, the output 
continues to consist of the combination of radio program at 
it's initial level and the sounds from the baby's room. The 
period between T 2 and T 3 the situation is identical to that 
between T 0 and T x . However, between T 3 and T 4 , the noise in 
the baby's room is above the threshold value B T . In response 
to audio source B crossing the threshold, the system brings 
the noise in the baby's room to the foreground by reducing 
the level of mixer input A to a level A A for the period during 
which the noise in the baby's room is above the threshold. In 
the limiting case, the level A A would be zero. 

The previously discussed scenarios have dealt with the 
combination of two continuous audio sources and the 
combination of a continuous audio source and signal event 
source. The remaining combination of two signal event sources 
is the least probable situation due to the short duration of 
signal events, but requires a different approach for 
balancing. Signal events are typically a single pulse at a 
given tone, a sequence of pulses at a single tone, or a 
sequence of pulses of different tones. If a situation arises 
where two signal events overlap, the second event can be 
stored by a buffer in the priority logic unit at the same 
time it is being suppressed by its associated variable 
attenuator/amplifier. After the first signal event has ended, 
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the priority logic unit can replay the stored signal event so 
that both signal events may be heard and overlap is avoided. 



It can be seen from the present embodiments that 
5 multiple audio sources in a handheld device can be 

prioritized and a balanced output obtained based upon the 
prioritization. The capability for balancing combined audio 
sources in a single output offers more flexibility than 
simply choosing between multiple sources or combining them 
10 without prioritization. 



% The foregoing descriptions of specific embodiments of 

the present invention have been presented for purposes of 
j illustration and description. They are not intended to be 
Ul 15 exhaustive or to limit the invention to the precise forms 
Q disclosed, and obviously many modifications and variations 
yj are possible in light of the above teaching. The embodiments 
g were chosen and described in order to best explain the 

principles of the invention and its practical application, to 
20 thereby enable others skilled in the art to best utilize the 
invention and various embodiments with various modifications 
are suited to the particular use contemplated. It is 
intended that the scope of the invention be defined by the 
Claims appended hereto and their equivalents. 
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